Wearable electronic device controlling noise canceling of external wearable electronic device and method of operating the same

ABSTRACT

According to certain embodiments, an electronic device comprises: at least one display module; at least one communication module; at least one microphone; at least one camera; and a processor, wherein the processor is configured to: capture at least one image through the at least one camera, identify at least one object located around the electronic device, based at least in part on the at least one image, identify sound information attributable to the identified at least one object from first sound data input through the at least one microphone, control the at least one display module to display at least one virtual object corresponding to the identified at least one object at a position corresponding to the at least one object, wherein the position is determined based on the sound information, obtain a first user input associated with a first virtual object among the at least one virtual object, and determine a noise cancellation (NC) level of a first object corresponding to the first virtual object, based on the first user input.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of International Application No.PCT/KR2021/020316 designating the United States, filed on Dec. 30, 2021,in the Korean Intellectual Property Receiving Office and claimingpriority to Korean Patent Application No. 10-2021-0048034, filed on Apr.13, 2021, in the Korean Intellectual Property Office, the disclosures ofwhich are incorporated by reference herein in their entireties.

BACKGROUND 1. Field

Certain embodiments of the disclosure relate to a wearable electronicdevice for controlling noise canceling of an external wearableelectronic device, and a method of operating the same.

2. Description of Related Art

Augmented reality (AR) makes it possible for the user to see a realenvironment, and providing a better sense of reality through additionalinformation. The user can observe the image together with a realenvironment to identify, for example, information about an object in thecurrently observed environment.

An AR device may be a wearable electronic device. For example, anelectronic device in the form of AR glasses that can be worn on the facelike glasses is becoming widespread.

However, the sound on an AR device often includes a mixture of virtualsounds and ambient noise. The ambient noise can be distracting anddeteriorate the user experience.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

According to certain embodiments, an electronic device comprises: atleast one display module; at least one communication module; at leastone microphone; at least one camera; and a processor, wherein theprocessor is configured to: capture at least one image through the atleast one camera, identify at least one object located around theelectronic device, based at least in part on the at least one image,identify sound information attributable to the identified at least oneobject from first sound data input through the at least one microphone,control the at least one display module to display at least one virtualobject corresponding to the identified at least one object at a positioncorresponding to the at least one object, wherein the position isdetermined based on the sound information, obtain a first user inputassociated with a first virtual object among the at least one virtualobject, and determine a noise cancellation (NC) level of a first objectcorresponding to the first virtual object, based on the first userinput.

According to certain embodiments, a method of operating an electronicdevice comprises: capturing at least one image through at least onecamera of the electronic device, identifying at least one object locatedaround the electronic device, based at least in part on the at least oneimage, identifying sound information attributable to the identified atleast one object from first sound data input through at least onemicrophone of the electronic device, displaying at least one virtualobject corresponding to the at least one object at a positioncorresponding to the identified at least one object through at least onedisplay module of the electronic device, wherein the position isdetermined based on the sound information, obtaining a first user inputassociated with a first virtual object among the at least one virtualobject, and determining a noise cancellation (NC) level of a firstobject corresponding to the first virtual object, based on the firstuser input.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the structure of a wearable electronic device,according to certain embodiments.

FIG. 2 illustrates the structures of a display and an eye trackingcamera of a wearable electronic device, according to certainembodiments.

FIG. 3 illustrates a block diagram of a wearable electronic device,according to certain embodiments.

FIG. 4 is a block diagram of an external wearable electronic device,according to certain embodiments.

FIG. 5 is a block diagram of electronic devices in a networkenvironment, according to certain embodiments.

FIG. 6 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments.

FIG. 7 is a flowchart illustrating a method of operating a wearableelectronic device according to certain embodiments.

FIG. 8 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments.

FIG. 9 is a diagram illustrating a method of operating a wearableelectronic device.

FIG. 10 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments.

FIG. 11 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments.

FIG. 12 is a diagram illustrating a method of operating a wearableelectronic device, according to certain embodiments.

FIG. 13 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments.

FIG. 14 is a flowchart illustrating a method of operating a wearableelectronic device according to certain embodiments.

FIG. 15 is a diagram illustrating a method of operating an externalelectronic device, according to certain embodiments.

FIG. 16 is a diagram illustrating a method of operating a wearableelectronic device, according to certain embodiments.

FIG. 17 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments.

FIG. 18 is a diagram illustrating a method of operating a wearableelectronic device.

FIG. 19 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments.

FIG. 20 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments.

FIG. 21 is a diagram illustrating a method of operating a wearableelectronic device.

FIG. 22 is a block diagram of an electronic device in a networkenvironment, according to certain embodiments.

DETAILED DESCRIPTION

Noise canceling (NC) is a technology for canceling out or blockingexternal noise that interferes with hearing sound from an audio device.The noise cancellation technology includes an active noise canceling(ANC) method and a passive noise canceling (PNC) method. The activenoise cancellation method is a method of analyzing the wave of externalnoise and generating a wave having the opposite phase of the wave tocancel the noise. The passive noise cancellation method is a method ofphysically blocking noise.

A noise cancellation device may be a wearable electronic device. Forexample, noise cancellation devices that can be worn on or near the ear,such as headsets or earphones, have become widespread.

According to certain embodiments of the disclosure, it is possible toprovide an augmented reality device including a method for controlling areal sound using a noise cancellation device in an augmented realityenvironment.

According to certain embodiments, by providing a wearable electronicdevice for controlling noise canceling of an external wearableelectronic device and a method of operating the same, it is possible tocontrol a virtual sound and a real sound in an augmented realityenvironment, which makes it possible for the user to experience arealistic augmented reality environment.

FIG. 1 illustrates the structure of a wearable electronic device 100.The wearable electronic device 100, which will be explained in moredetail below, incorporates augmented reality. Augmented reality makes itpossible for the user to see a real environment, and providing a bettersense of reality through additional information. The wearable electronicdevice 100 includes cameras 111-1, 111-2, 112-1, 112-2, and/or 113 forcapturing the real environment surrounding the user. The realenvironment surrounding the user can be displayed on the displays 151,152, as well as additional information regarding the real environment.

Additionally, the user can also hear virtual sounds through speakers163-1 and 163-2. However, the sound often includes a mixture of virtualsounds and ambient noise. The ambient noise can be distracting anddeteriorate the user experience. Accordingly, the wearable electronicdevice 100 includes uses noise cancellation.

FIG. 1 illustrates the structure of a wearable electronic device,according to certain embodiments. The wearable electronic device 100 mayinclude a frame 105, a first support 101, a second support 102, a firsthinge 103 connecting the frame 105 with the first support 101, and/or asecond hinge 104 connecting the frame 105 with the second support 102.

The frame 105 may also include one or more first cameras 111-1 and111-2, one or more second cameras 112-1 and 112-2, and/or a third camera113, one or more light-emitting elements 114-1 and 114-2, a firstdisplay 151 and a second display 152, one or more sound input devices162-1, 162-2 and 162-3, or one or more transparent members 190-1 and190-2.

An image obtained through the one or more first cameras 111-1 and 111-2may be used for detecting a hand gesture by a user, tracking a user'shead, and/or recognizing a space. The one or more first cameras 111-1and 111-2 may be GS (global shutter) cameras. The one or more firstcameras 111-1 and 111-2 may perform a SLAM (simultaneous localizationand mapping) operation through depth photographing. The one or morefirst cameras 111-1 and 111-2 may perform spatial recognition for 6 DoF(degrees of freedom).

An image obtained through the one or more second cameras 112-1 and 112-2may be used to detect and track the user's pupils. For example, an imageobtained through the one or more second cameras 112-1 and 112-2 may beused to track the user gaze direction. The one or more second cameras112-1 and 112-2 may be GS cameras. The one or more second cameras 112-1and 112-2 may respectively correspond to the left eye and the right eye,and the performance of the one or more second cameras 112-1 and 112-2may be substantially the same.

The at least one third camera 113 may be a high-resolution camera. Theat least one third camera 113 may execute an auto-focusing (AF) functionor a shake correction function. The at least one third camera 113 may bea GS camera or an RS (rolling shutter) camera.

The wearable electronic device 100 may include one or morelight-emitting elements 114-1 and 114-2. The light-emitting elements114-1 and 114-2 are different from a light source, which will bedescribed later, for emitting light to a screen output area of thedisplay. The light-emitting elements 114-1 and 114-2 may emit light tofacilitate detecting of the pupils in detecting and tracking of theuser's pupils through the one or more second cameras 112-1 and 112-2.The light-emitting elements 114-1 and 114-2 may include LEDs. Thelight-emitting elements 114-1 and 114-2 may emit infrared rays. Thelight-emitting elements 114-1 and 114-2 may be attached to the peripheryof the frame 105 of the wearable electronic device 100. Thelight-emitting elements 114-1 and 114-2 may be located in the vicinityof the one or more first cameras 111-1 and 111-2, and assist the one ormore first cameras 111-1 and 111-2 in gesture detection, head tracking,and/or spatial recognition when the wearable electronic device 100 isused in a dark environment. The light-emitting elements 114-1 and 114-2may be located in the vicinity of the at least one third camera 113 andassist the at least one third camera 113 in obtaining images when thewearable electronic device 100 is used in a dark environment.

The wearable electronic device 100 may include a first display 151, asecond display 152, one or more optical input members 153-1 and 153-2,one or more transparent members 190-1 and 190-2, and/or one or morescreen display regions 154-1 and 154-2, which are positioned in theframe 105. The first display 151 and the second display 152 may include,for example, a liquid crystal display (LCD), a digital mirror device(DMD), a liquid crystal-on-silicon (LCoS) display, an organiclight-emitting diode (OLED), or a micro light-emitting diode (microLED). According to certain embodiments, in the case where the firstdisplay 151 and the second display 152 are configured as one of a liquidcrystal display device, a digital mirror device, or a liquidcrystal-on-silicon display, the wearable electronic device 100 mayinclude a light source for emitting light to a screen output area of thedisplay. According to other certain embodiments, in the case where thefirst display 151 and/or the second display 152 are able to emit lightby themselves, for example, in the case where they are configured as oneof an organic light-emitting diode or a micro LED, it is possible toprovide a virtual image of good quality to the user even if the wearableelectronic device 100 does not include a separate light source.

The one or more transparent members 190-1 and 190-2 may be disposed toface the user's eyes when the user wears the wearable electronic device100. The one or more transparent members 190-1 and 190-2 may include atleast one of a glass plate, a plastic plate, and a polymer. According tocertain embodiments, when the user wears the wearable electronic device100, the user can see the external world through the one or moretransparent members 190-1 and 190-2. According to certain embodiments,one or more optical input members 153-1 and 153-2 may guide the lightgenerated by the first display 151 and/or the second display 152 to theuser's eyes. An image based on the light generated by the first display151 and the second display 152 may be formed on the one or more screendisplay regions 154-1 and 154-2 on the one or more transparent members190-1 and 190-2, and the user may see the image formed on the one ormore screen display regions 154-1 and 154-2.

The wearable electronic device 100 may include one or more opticalwaveguides (not shown). The optical waveguide may transmit the lightgenerated by the first display 151 and the second display 152 to theuser's eyes. The wearable electronic device 100 may include opticalwaveguides that respectively correspond to the left eye and the righteye. The optical waveguide may include at least one of glass, plastic,or polymer. The optical waveguide may include a nano-pattern, forexample, a polygonal or curved grated structure, formed on an internalor external surface. The optical waveguide may include a free-form typeprism, and in this case, the optical waveguide may provide incidentlight to the user through a reflection mirror. The optical waveguide mayinclude at least one of at least one diffractive element (e.g., a DOE(diffractive optical element) or a HOE (holographic optical element)) ora reflective element (e.g., a reflection mirror), and guide the displaylight emitted from a light source to the user's eyes using the at leastone diffractive element or reflective element included therein. Thediffractive element may include an optical input/output member. Thereflective element may include a member causing total reflection.

The wearable electronic device 100 may include one or more sound inputdevices 162-1, 162-2, and 162-3 (e.g., microphones), and the one or moresound input devices 162-1, 162-2, and 162-3 may receive an input soundof the user or an input sound generated in the vicinity of the wearableelectronic device 100. For example, the one or more sound input devices162-1, 162-2, and 162-3 may receive an input sound generated in thevicinity and transmit the same to a processor (e.g., the processor 320in FIG. 3).

According to certain embodiments, one or more supports (e.g., the firstsupport 101 and the second support 102) may include PCBs (e.g., a firstPCB 170-1 and a second PCB 170-2), one or more sound output devices163-1 and 163-2, and one or more batteries 135-1 and 135-2. The firstPCB 170-1 and the second PCB 170-2 may transmit electrical signals toelements included in the wearable electronic device 100, such as a firstcamera 211, a second camera 212, a third camera 213, a display module250, an audio module 261, and/or a sensor 280, which will be describedlater with reference to FIG. 2. The first PCB 170-1 and the second PCB170-2 may be FPCBs. According to certain embodiments, each of the firstPCB 170-1 and the second PCB 170-2 may include a first substrate, asecond substrate, and an interposer disposed between the first substrateand the second substrate. The wearable electronic device 100 may includebatteries 135-1 and 135-2. The batteries 135-1 and 135-2 may store powerfor operating the remaining elements of the wearable electronic device100. The one or more sound output devices 163-1 and 163-2 (e.g.,speakers) may output audio data to the user. For example, it is possibleto provide feedback on a user's command (or input) or to provideinformation on a virtual object to the user through audio data.

The wearable electronic device 100 may include one or more hinges (e.g.,the first hinge 103 and the second hinge 104). For example, the firsthinge 103 may couple the first support 101 to the frame 105 so as to berotatable with respect to the frame 105, and the second hinge 104 maycouple the second support 102 to the frame 105 so as to be rotatablewith respect to the frame 105.

As noted above, the wearable electronic device 100 uses noisecancellation to remove or reduce ambient noise. The wearable electronicdevice 100 uses one of the camera 111-1, 111-2, 112-1, 112-2, and/or 113to obtain image(s) of the environment surrounding the user, as well asinformation about the surrounding environment from an external source.The wearable electronic device 100 uses the foregoing to identifyobject(s) in the surrounding environment.

With the identification of object(s) in the surrounding environments,the electronic wearable device 100 can obtain sound data regarding theobject(s), and then identify sound from the microphones 162-1, 162-2,and 162-3 that corresponds to the object(s) and cancel it.

FIG. 2 illustrates the structures of a display and an eye trackingcamera of a wearable electronic device, according to certainembodiments. The wearable electronic device 100 may include a display221, an optical input member 222, an optical display waveguide 223, anoptical output member 224, an eye tracking camera 210, a first splitter241, an optical eye tracking waveguide 242, and/or a second splitter243.

In the wearable electronic device 100, the display 221 may be one of thefirst display 151 or the second display 152 shown in FIG. 1. The lightoutput from the display 221 may pass through the optical input member222 (e.g., the optical input members 153-1 and 153-2 in FIG. 1) to beincident on the optical display waveguide 223, and may be output throughthe optical output member 224 via the optical display waveguide 223. Thelight output from the optical output member 224 may be seen by the userwith an eye 230. Hereinafter, in this specification, the expression“display an object on the display” may indicate that the light outputfrom the display 221 is output through the optical output member 224 andthat the shape of the object is seen in the eye 230 of the user by thelight output through the optical output member 224. In addition, theexpression “control the display to display an object” may indicate thatthe display 221 is controlled such that the light output from thedisplay 221 is output through the optical output member 224 and suchthat the shape of the object is seen in the eye 230 of the user.

The light 235 reflected from the eye 230 of the user may pass throughthe first splitter 241 to be incident on the optical eye trackingwaveguide 242, and pass through the optical eye tracking waveguide 242to be output to the eye tracking camera 210 through the second splitter243. The light 235 reflected from the eye 230 of the user may be thelight output from the light-emitting elements 114-1 and 114-2 in FIG. 1and reflected from the eye 230 of the user. The eye tracking camera 210may be the one or more second cameras 112-1 and 112-2 shown in FIG. 1.

FIG. 3 illustrates a block diagram of a wearable electronic device,according to certain embodiments.

Referring to FIG. 3, according to certain embodiments, the wearableelectronic device 100 may include a first camera 311, a second camera312, a third camera 313, a processor 320, a PMIC 330, a battery 335, amemory 340, a display module 350, an audio module 361, a sound inputdevice 362, a sound output device 363, communication module 370, and/ora sensor 380.

According to certain embodiments, details of the one or more firstcameras 111-1 and 111-2, the one or more second camera 112-1 and 112-2,and at least one third camera 113, which have been described above withreference to FIG. 1, may be applied to each of the first camera 311, thesecond camera 312, and the third camera 313 in substantially the samemanner. The wearable electronic device 100 may include a plurality of atleast one of the first cameras 311, the second cameras 312, and thethird cameras 313.

The processor 320 may control other elements of the wearable electronicdevice 100, for example, the first camera 311, the second camera 312,the third camera 313, and the PMIC 330, the memory 340, the displaymodule 350, the audio module 361, the communication module 370, and/orthe sensor 380, and perform processing of a variety of data oroperations.

The PMIC 330 may convert power stored in the battery 335 to have acurrent or voltage required by other elements of the wearable electronicdevice 100 and supply the same to other elements of the wearableelectronic device 100.

The memory 340 may store a variety of data used by at least one element(e.g., the processor 320 or the sensor module 380) of the wearableelectronic device 100.

The display module 350 may display a screen to be provided to the user.The display module 350 may include the first display 151, the seconddisplay 152, the one or more optical input members 153-1 and 153-2, theone or more transparent members 190-1 and 190-2, and/or the one or morescreen display regions 154-1 and 154-2, which have been described abovewith reference to FIG. 1.

The audio module 361 may be connected to the sound input device 362 andthe sound output device 363, convert data input through the sound inputdevice 362, and may convert the data to be output to the sound outputdevice 363. The sound output device 363 may include a speaker and anamplifier. The sound output device 363 may indicate the one or moresound output devices 163-1 and 163-2 in FIG. 1. The sound input device362 may indicate one or more sound input devices 162-1, 162-2, and 162-3in FIG. 1.

The communication module 370 may support establishment of a wirelesscommunication channel with an electronic device outside the wearableelectronic device 100 and communication through the establishedcommunication channel.

The sensor 380 may include a 6-axis sensor 381, a magnetic sensor 382, aproximity sensor 383, and/or an optical sensor 384. The sensor 380 mayinclude a sensor for obtaining a biometric signal for detecting whetheror not the wearable electronic device 100 is worn by the user. Forexample, the sensor 380 may include at least one of a heart rate sensor,a skin sensor, and a temperature sensor.

The processor 320 may identify movement of the user wearing the wearableelectronic device 100 through the 6-axis sensor 381. For example, the6-axis sensor 381 may detect a change in the direction in which the userfaces (e.g., the direction in which the user views through the wearableelectronic device 100) to generate a sensor value, and transmit thegenerated sensor value or the amount of change in the sensor value tothe processor 320.

The audio module 361 may receive a sound generated in the vicinity ofthe wearable electronic device 100 (or the user) through the sound inputdevice 362, and transmit data converted from the received sound to theprocessor 320.

The communication module 370 may transmit/receive data to and from anexternal electronic device (e.g., a wearable electronic device (e.g.,earphones) or an external electronic device (e.g., a terminal)). Forexample, the wearable electronic device 100 may receive audio datareceived by the external wearable electronic device through thecommunication module 370, and transmit the received audio data to theprocessor 320. As another example, the wearable electronic device 100may output, through the display module 350, image data based on the datareceived from the external electronic device through the communicationmodule 370.

FIG. 4 is a block diagram of an external wearable electronic device,according to certain embodiments. The external wearable electronicdevice 410 may be a wearable electronic device in the form of earphones.The external wearable electronic device 410 may have a plurality ofphysically separated housings. For example, in the case where theexternal wearable electronic device 410 is a wearable electronic devicein the form of earphones, the external wearable electronic device 410may include a first housing to be worn on the left ear and a secondhousing to be worn on the right ear, and in this case, the respectiveelements shown in FIG. 4 may be included in one or more of the pluralityof housings. In certain embodiments, the external wearable electronicdevice 410 can correspond to sound output devices 163-1 and 163-2.

The external wearable electronic device 410 may include a processor 415,a memory 420, a communication module 430, an audio module 440, a sensormodule 450, and/or a battery 460.

The processor 415 may receive data from other elements of the externalwearable electronic device 410, for example, the memory 420, thecommunication module 430, the audio module 440, the sensor module 450,and/or the battery 460, perform operations based on the received data,and transmit signals for controlling other elements to other elements.The processor 415 may operate based on instructions stored in the memory420.

The memory 420 may store instructions that cause other elements of theexternal wearable electronic device 410, for example, the processor 415,the communication module 430, the audio module 440, the sensor module450, and/or the battery 460 to perform specified operations. The memory420 may store audio data obtained through the audio module 440.

The communication module 430 may perform wireless communication withanother electronic device (e.g., the wearable electronic device 100).The communication module 430 may transmit information obtained from theexternal wearable electronic device 410 to the wearable electronicdevice 100. As another example, the communication module 430 may receiveaudio data from the wearable electronic device 100. The type ofcommunication supported by the communication module 430 is not limited.

The audio module 440 may include a plurality of microphones and/or oneor more speakers. The plurality of microphones may include a microphonefacing in the direction toward the user's inner ear when the user wearsthe external wearable electronic device 410 and a microphone facing inthe direction away from the user when the user wears the externalwearable electronic device 410. The audio module 440 may obtain audiodata through the plurality of microphones, respectively, and performnoise canceling, based on the audio data obtained through the pluralityof microphones. The audio module 440 may output audio data obtained fromanother electronic device (e.g., the wearable electronic device 100)through the communication module 430. As another example, the audiomodule 440 may perform noise canceling, based on audio data obtainedfrom another electronic device (e.g., the wearable electronic device100) through the communication module 430.

The sensor module 450 may include a biometric sensor for detectingwhether or not the user wears the external wearable electronic device410. For example, the biometric sensor may include at least one of aheart rate sensor, a skin sensor, and a temperature sensor. The sensormodule 450 may include a geomagnetic sensor.

The external wearable electronic device 410 may receive a request fordata transmission from the wearable electronic device 100 through thecommunication module 430. For example, the external wearable electronicdevice 410 may receive a request for transmission of audio data receivedthrough the audio module 440.

FIG. 5 is a block diagram of electronic devices in a networkenvironment, according to certain embodiments. Referring to FIG. 5, thenetwork environment may include a wearable electronic device 100, anexternal wearable electronic device 410, an external electronic device520 (e.g., the electronic device 2201 to be described later withreference to FIG. 22), and/or a server 530 (e.g., the server 2208 to bedescribed later with reference to FIG. 22).

Referring to FIG. 5, according to certain embodiments, the wearableelectronic device 100 may communicate with the external wearableelectronic device 410. The wearable electronic device 100 maycommunicate with the external electronic device 520. The externalelectronic device 520 may communicate with the external wearableelectronic device 410. The wearable electronic device 100 may performindirect communication with the external wearable electronic device 410through the external electronic device 520. The external electronicdevice 520 may communicate with the server 530. The wearable electronicdevice 100 may perform indirect communication with the server 530through the external electronic device 520. The wearable electronicdevice 100 may directly communicate with the server 530. Referring toFIG. 5, electronic devices in the network environment may communicatewith each other, and there is no limitation as to the communicationmethod. Although the wearable electronic device 100 and the externalwearable electronic device 410 are shown as separate, it shall beunderstood that in some embodiments, the wearable electronic device 100and the external wearable electronic device 410 can be integrated into asingle device. Thus, communication between the wearable electronicdevice 100 and the external wearable electronic device 410 may bethrough internal circuitry.

FIG. 6 is a flowchart illustrating a method of operating a wearableelectronic device according to certain embodiments.

Referring to FIG. 6, in operation 601, according to certain embodiments,a wearable electronic device 100 (e.g., the processor 320) may obtain atleast one image input through at least one camera (e.g., the firstcamera 311, the second camera 312, and/or the third camera 313). Forexample, the wearable electronic device 100 may obtain at least oneimage including at least one object located in the vicinity of thewearable electronic device 100 through at least one camera.

In operation 603, according to certain embodiments, the wearableelectronic device 100 may obtain surrounding environment informationfrom an external device (e.g., the external electronic device 520 and/orthe server 530) through the communication module 370. For example, thesurrounding environment information may include position information,sound information, and/or state information of at least one objectlocated in the vicinity of the wearable electronic device 100. Forexample, the position information of at least one object may includeglobal positioning system (GPS) coordinate information of at least oneobject and/or relative coordinate information between the wearableelectronic device 100 and at least one object (e.g., distanceinformation or direction information). For example, the soundinformation of at least one object may include information on the typeof sound, the volume of sound, the frequency, waveform, and amplitude ofsound, and/or the direction of sound of at least one object. Forexample, the state information of at least one object may include anoperation state of at least one object (e.g., a speaker output state ofan external speaker or a power operation state of a peripheral homeappliance). The wearable electronic device 100 may transmit currentposition information of the wearable electronic device 100 to anexternal device (e.g., the external electronic device 520 and/or theserver 530) through the communication module 370, and obtain surroundingenvironment information about an object (e.g., an external speaker, aperipheral home appliance, a surrounding construction site, an externalperformance hall, a surrounding airport, or a surrounding road) locatedin the vicinity of the wearable electronic device 100 from the externaldevice.

In operation 605, according to certain embodiments, the wearableelectronic device 100 may identify at least one object (hereinafter, asurrounding object) located in the vicinity of the wearable electronicdevice 100, based on at least one image (hereinafter, an input image)input through at least one camera (e.g., the first camera 311, thesecond camera 312, and/or the third camera 313) in operation 601, and/orsurrounding environment information (hereinafter, surroundingenvironment information) obtained from the external device (e.g., theexternal electronic device 520 and/or server 530) through thecommunication module 370 in operation 603. For example, the wearableelectronic device 100 may identify surrounding objects using only aninput image. For example, the wearable electronic device 100 mayidentify surrounding objects included in the input image even in thecase of failing to obtain the surrounding environment information. Asanother example, the wearable electronic device 100 may identifysurrounding objects using only surrounding environment information. Forexample, even when an input image is not received, the wearableelectronic device 100 may identify surrounding objects, based onsurrounding environment information. As another example, the wearableelectronic device 100 may identify surrounding objects, based on aninput image and surrounding environment information. For example, thewearable electronic device 100 may identify both the surrounding objectsincluded in the input image and the surrounding objects identified basedon the surrounding environment information. As described above, any oneof operation 601 or operation 603 may be omitted.

For example, if user is in their kitchen and there is a blender that isin operation, the blender will generate considerable noise. The cameracan obtain a picture that includes the blender during operation 601.During operation 603, the wearable electronic device 100 may determinethat the user's location is in their kitchen using GPS coordinates.Moreover, the wearable electronic device 100 may also receiveinformation of appliances that are commonly found in the kitchen, suchas blenders, dishwashers, microwave ovens, etc., as well as theirtypical noise patterns. During operation 605, the wearable electronicdevice 100 can determine that the image includes a blender.

According to certain embodiments, if a surrounding object identifiedbased on the surrounding environment information obtained from anexternal device (e.g., the external electronic device 520 and/or theserver 530) is not included in at least one image input through at leastone camera (e.g., the first camera 311, the second camera 312, and/orthe third camera 313), the wearable electronic device 100 may display,through the display module 350, a virtual object corresponding to asurrounding object that is identified based on the surroundingenvironment information (e.g., if the surrounding object is an airport,an icon corresponding to the airport) at a corresponding positionidentified based on the surrounding environment information. At thistime, if the surrounding object identified based on the surroundingenvironment information is located outside a display range of thedisplay module 350 of the wearable electronic device 100, the wearableelectronic device 100 may display a virtual object indicating theposition corresponding to the surrounding object (e.g., an arrow objectpointing to the position corresponding to the surrounding object).

For example, if there is an object of note that is behind the user'sback, the wearable electronic device 100 can place a virtual objectindicating its relative presence to the user. If, for instance, behindthe user to the right, there is an exit, the wearable electronic device100 can place a symbol for an exit door with a notation “4:00”indicating the direction from the user).

FIG. 7 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments.

Referring to FIG. 7, in operation 701, according to certain embodiments,the wearable electronic device 100 (e.g., the processor 320) may obtainfirst sound data input through at least one microphone (e.g., the soundinput device 362 in FIG. 3). For example, the wearable electronic device100 may obtain first sound data converted from a sound from at least oneobject located in the vicinity of the wearable electronic device 100through at least one microphone.

In operation 703, according to certain embodiments, the wearableelectronic device 100 may obtain second sound data from an externaldevice (e.g., the external wearable electronic device 410 and/or theexternal electronic device 520) through the communication module 370.For example, the external wearable electronic device 410 may obtainsecond sound data converted from a sound from at least one objectlocated in the vicinity thereof, and the wearable electronic device 100may receive the second sound data from the external wearable electronicdevice 410 through the communication module 370.

In operation 705, according to certain embodiments, based on the firstsound data input through the at least one microphone in operation 701and/or the second sound data obtained from the external device (e.g.,the external wearable electronic device 410 and/or the externalelectronic device 520) through the communication module 370 in operation703, the wearable electronic device 100 may identify sound information(e.g., information about the type of sound, the volume of sound, thefrequency, waveform, and amplitude of sound, and/or the direction ofsound corresponding to each of at least one object) corresponding to atleast one object located in the vicinity of the wearable electronicdevice 100 (hereinafter, surrounding objects).

For example, the wearable electronic device 100 may identify soundinformation of surrounding objects using only the first sound data. Asanother example, the wearable electronic device 100 may identify soundinformation of surrounding objects only using the second sound data. Asanother example, the wearable electronic device 100 may identify soundinformation of surrounding objects, based on the first sound data andthe second sound data. For example, the wearable electronic device 100may obtain third sound data by processing the first sound data, based onthe second sound data, and identify sound information of eachsurrounding object, based on the third sound data. For example, theprocess for obtaining the third sound data may be a noise cancellationprocess for removing ambient noise except for a sound corresponding tothe surrounding objects. As described above, any one of operation 701 oroperation 703 may be omitted.

In the example where the user is in their kitchen where there is ablender, the microphone can pick up the sound from the blender duringits operation during operation 701. During operation 703, the wearableelectronic device 100 can obtain information about the soundcharacteristics of a blender and during operation 705, the wearableelectronic device 100 can determine that portion of the signal picked upby the microphone that is attributable to the blender.

FIG. 8 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments. FIG. 8 will bedescribed with reference to FIG. 9. FIG. 9 is a diagram illustrating amethod of operating a wearable electronic device.

Referring to FIG. 8, in operation 801, according to certain embodiments,the wearable electronic device 100 (e.g., the processor 320), based onsound information (e.g., the sound information included in thesurrounding environment information obtained in operation 603 in FIG. 6and/or the sound information identified in operation 705 in FIG. 7), maydisplay, through the display module 350 of the wearable electronicdevice 100, at least one virtual object corresponding to at least oneobject at a position corresponding to the at least one object located inthe vicinity of the wearable electronic device 100.

For example, referring to FIG. 9, the wearable electronic device 100 maydisplay an augmented reality screen 900 through the display module 350.The user may recognize at least one object (e.g., a first object 910corresponding to a person and/or a second object 920 corresponding to anexternal speaker) located in the vicinity thereof directly through theeyes of the user while wearing the wearable electronic device 100. Theuser may recognize the augmented reality screen 900 displayed throughthe display module 350 of the wearable electronic device 100.Accordingly, the user may recognize objects existing in reality (e.g.,the first object 910 and/or the second object 920) and virtual objects(e.g., the first virtual object 911 and/or the second virtual object921) displayed on the augmented reality screen 900 while wearing thewearable electronic device 100. The wearable electronic device 100 maydisplay the first virtual object 911 corresponding to the first object910 at a position corresponding to the first object (e.g., the areanearby the first object 910), and display the second virtual object 921corresponding to the second object 920 at a position corresponding tothe second object (e.g., the area nearby the second object 920).

For example, referring to FIG. 9, the first virtual object 911 and/orthe second virtual object 921 may include virtual objects (e.g., 912 and922) indicating whether or not to block noise and/or virtual objects(e.g., 914 and 924) for adjusting the noise volume. The virtual object912, which is included in the first virtual object 911, for indicatingwhether or not to block noise may be a virtual object indicating whetheror not to block the sound from the first object 910 corresponding to thefirst virtual object 911, and the virtual object 922, which is includedin the second virtual object 921, for indicating whether or not to blocknoise may be a virtual object indicating whether or not to the soundfrom the second object 920 corresponding to the second virtual object921. The virtual object 914 for noise volume adjustment included in thefirst virtual object 911 may be a virtual object indicating the degreeof adjusting the output volume of the sound from the first object 910corresponding to the first virtual object 911, and the virtual object924 for noise volume adjustment included in the second virtual object921 may be a virtual object indicating the degree of adjusting theoutput volume of the sound from the second object 920 corresponding tothe second virtual object 921.

The wearable electronic device 100 (e.g., the processor 320) maydetermine whether or not to display a virtual object (e.g., the firstvirtual object 911) corresponding to a specific object (e.g., the firstobject 910) among the at least one object located in the vicinity of thewearable electronic device 100 (e.g., the first object 910 or the secondobject 920 in FIG. 9, or surrounding objects identified based onsurrounding environment information obtained from an external device(e.g., the external electronic device 520 and/or the server 530)), basedon the volume of sound included in the sound information correspondingto the specific object (e.g., the first object 910), and/or whether thevolume of the sound exceeds a threshold.

The wearable electronic device 100 may determine to display a virtualobject (e.g., the first virtual object 911) corresponding to a specificobject (e.g., the first object 910), based on that the volume of soundincluded in the sound information corresponding to the specific object(e.g., the first object 910) is greater than or equal to a predeterminedfirst volume or is less than or equal to a predetermined second volume.At this time, the second volume may be less than the first volume. Forexample, referring to FIG. 9, the wearable electronic device 100 maydetermine to display the first virtual object 911 corresponding to thefirst object 910 and the second virtual object 921 corresponding to thesecond object 920, based on that the volume of sound included in soundinformation corresponding to the first object 910 is less than or equalto a predetermined second volume and that the volume of sound includedin sound information corresponding to the second object 920 is greaterthan or equal to a predetermined first volume. Alternatively, althoughnot shown, the wearable electronic device 100 may determine to displaythe first virtual object 911 corresponding to the first object 910 andto not display the second virtual object 921 corresponding to the secondobject 920, based on that the volume of sound included in soundinformation corresponding to the first object 910 is less than or equalto a predetermined second volume and that the volume of sound includedin sound information corresponding to the second object 920 is less thanor equal to a predetermined first volume and greater than or equal to apredetermined second volume.

In operation 803, according to certain embodiments, the wearableelectronic device 100 may obtain a user input (e.g., a first user input)associated with a virtual object (e.g., the first virtual object 911)among the at least one virtual object (e.g., 911 and 921). For example,the wearable electronic device 100 may obtain a user gesture input(e.g., an input for selecting a specific virtual object (e.g., 912)displayed on the augmented reality screen 900 by moving the user's handto a specific position) through at least one camera (e.g., the firstcamera 311, the second camera 312, and/or the third camera 313). Asanother example, the wearable electronic device 100 may obtain a usergaze input (e.g., an input for selecting a virtual object (e.g., 912)corresponding to the user gaze direction from among the virtual objectsdisplayed on the augmented reality screen 900) by tracking the eye ofthe user through at least one camera (e.g., the first camera 311, thesecond camera 312, and/or the third camera 313). As another example, thewearable electronic device 100 may obtain a user utterance input (e.g.,an input for selecting a virtual object (e.g., 912) corresponding to theuser's utterance) through at least one microphone (e.g., the sound inputdevice 362 in FIG. 3). As another example, the wearable electronicdevice 100 may obtain a user input through a separate input device(e.g., an input device (e.g., a keyboard device, a mouse device, a paddevice, or a hand-held device) for communicating with the wearableelectronic device 100 through the communication module 370, or an inputdevice included in the wearable electronic device 100 (e.g., a buttoninput device (not shown) or a touch input device (not shown) included inthe wearable electronic device 100)). For example, the wearableelectronic device 100 may obtain a user gesture input, a user gazeinput, a user utterance input, or a user input through a separate inputdevice for selecting the first virtual object 911 or a virtual object(e.g., 912 or 914) included in the first virtual object 911.

In operation 805, according to certain embodiments, the wearableelectronic device 100 may determine a noise cancellation level(hereinafter, an NC level) of an object (e.g., the first object 910)corresponding to a specific virtual object (e.g., first virtual object911), based on a user input (e.g., a first user input) associated withthe specific virtual object (e.g., the first virtual object 911) amongthe at least one virtual object (e.g., the first virtual object 911and/or the second virtual object 921) displayed on the augmented realityscreen 900. The NC level will be described in detail later withreference to FIG. 12.

In certain embodiments, the wearable electronic device 100 may determinewhether to display virtual objects corresponding to object in thevicinity based on the user's gaze. For example, it is possible that theuser is gazing at the object due to the noise that it is making.Accordingly, it would be helpful to provide virtual objects for noisecancellation.

FIG. 10 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments. FIG. 10 will bedescribed with reference to FIG. 9.

Referring to FIG. 10, in operation 1001, according to certainembodiments, the wearable electronic device 100 (e.g., the processor320) may identify a user gaze direction. For example, the wearableelectronic device 100 may identify a user gaze direction by tracking theeyes of the user through at least one camera (e.g., the first camera311, the second camera 312, and/or the third camera 313). The wearableelectronic device 100 may track the eyes of the user to identify a usergaze direction toward a specific object (e.g., the first object 910)among the at least one object located in the vicinity (e.g., the firstobject 910 or the second object 920 in FIG. 9, or virtual objectsdisplayed on the augmented reality screen 900 to correspond tosurrounding objects identified based on surrounding environmentinformation obtained from an external device (e.g., the externalelectronic device 520 and/or the server 530)).

In operation 1003, according to certain embodiments, the wearableelectronic device 100 may determine whether or not to display at leastone virtual object (e.g., 911 and 921 in FIG. 9), based on the user gazedirection. For example, the wearable electronic device 100 may determineto display a virtual object (e.g., first virtual object 911)corresponding to a specific object (e.g., first object 910)corresponding to the user gaze direction among the at least one object.

In this case, the wearable electronic device 100 may determine not todisplay a virtual object (e.g., the second virtual object 921)corresponding to at least another object (e.g., the second object 920)that does not correspond to the user gaze direction. As another example,the wearable electronic device 100 may determine not to be display avirtual object (e.g., the first virtual object 911) corresponding to aspecific object (e.g., the first object 910) corresponding to the usergaze direction, among the at least one object located in the vicinity ofthe wearable electronic device 100. In this case, the wearableelectronic device 100 may determine to display a virtual object (e.g.,the second virtual object 921) corresponding to at least another object(e.g., the second object 920) that does not correspond to the user gazedirection.

In certain embodiments, the wearable electronic device 100 may determinewhether to place virtual objects based on user gestures. The wearableelectronic device 100 can use a camera to observe the user's bodymotions. For example, if the user points in one direction, the wearableelectronic device 100 may determine what that the user has selected anobject that is in the vicinity in the pointed direction.

FIG. 11 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments. FIG. 11 will bedescribed with reference to FIG. 9.

Referring to FIG. 11, in operation 1101, according to certainembodiments, the wearable electronic device 100 (e.g., the processor320) may identify at least a part of the user's body. For example, thewearable electronic device 100 may track at least a part of the user'sbody through at least one camera (e.g., the first camera 311, the secondcamera 312, and/or the third camera 313) and identify the position of atleast a part of the user's body, thereby identifying a user gestureinput. For example, the wearable electronic device 100 may track atleast a part of the user's body to identify a user gesture input forselecting a specific object (e.g., the first object 910) from among theat least one object located in the vicinity of the wearable electronicdevice 100 (e.g., the first object 910 or the second object 920 in FIG.9, or virtual objects displayed on the augmented reality screen 900 tocorrespond to surrounding objects identified based on surroundingenvironment information obtained from an external device (e.g., theexternal electronic device 520 and/or the server 530)).

For example, the camera (e.g., the first camera 311, the second camera312, and/or the third camera 313) may look for the user's hands andattempt to determine whether the user is pointing in a particulardirection. The wearable electronic device 100 may determine that anobject that is in the particular direction is selected by the user.

In operation 1103, according to certain embodiments, the wearableelectronic device 100 may determine whether or not to display at leastone virtual object (e.g., 911 and 921 in FIG. 9), based on the positionof at least a part of the user's body. For example, the wearableelectronic device 100 may determine to display a virtual object (e.g.,the first virtual object 911) corresponding to a specific object (e.g.,the first object 910) corresponding to the user gesture input (e.g.,corresponding to the position of at least a part of the user's body),among the at least one object located in the vicinity of the wearableelectronic device 100 (e.g., the first object 910 or the second object920 in FIG. 9, or virtual objects corresponding to surrounding objectsidentified based on surrounding environment information obtained from anexternal device (e.g., the external electronic device 520 and/or theserver 530)). In this case, the wearable electronic device 100 maydetermine not to display a virtual object (e.g., the second virtualobject 921) corresponding to at least another object (e.g., the secondobject 920) that does not correspond to the user gesture input (e.g.,that does not correspond to the position of at least a part of theuser's body).

As another example, the wearable electronic device 100 may determine notto display a virtual object (e.g., the first virtual object 911)corresponding to a specific object (e.g., the first object 910) thatcorresponds to the user gesture input (e.g., that corresponds to theposition of at least a part of the user's body), among the at least oneobject located in the vicinity of the wearable electronic device 100. Inthis case, the wearable electronic device 100 may determine to display avirtual object (e.g., the second virtual object 921) corresponding to atleast another object (e.g., the second object 920) that does notcorrespond to the user gesture input (e.g., that does not correspond tothe position of at least a part of the user's body).

FIG. 12 is a diagram illustrating a method of operating a wearableelectronic device, according to certain embodiments.

Referring to FIG. 12, the wearable electronic device 100 (e.g., theprocessor 320) may determine each NC level corresponding to eachoperation. The levels (e.g., level 1 to level 4) in FIG. 12 are providedby way of example to classify respective operations, and are notlimited. The respective operations disclosed in FIG. 12 will bedescribed as follows.

For example, the operation of deactivating a noise cancellation (NC)function may indicate an operation of performing a function that allowsthe user to hear a real sound as it is, instead of performing a noisecancellation function on the real sound. For example, the wearableelectronic device 100 may determine an NC level (e.g., 100)corresponding to the operation of deactivating an NC function. Forexample, based on a user input (e.g., a user input for deactivating avirtual object (e.g., the virtual object 912 or the virtual object 922in FIG. 9)) to a virtual object (e.g., the virtual object 912 or thevirtual object 922 in FIG. 9) corresponding to the operation ofdeactivating a noise cancellation function, the wearable electronicdevice 100 may determine the NC level of an object (e.g., the firstobject 910 or the second object 920 in FIG. 9).

As another example, referring to FIG. 15 or FIG. 16 to be describedlater, the wearable electronic device 100 may determine the NC level ofthe wearable electronic device 100 to be an NC level of 100)corresponding to the operation of deactivating a noise cancellationfunction, based on a user input to a virtual object corresponding to thehearing aid operation for all surrounding noises (e.g., the virtualobject 1530 in FIG. 15 or a virtual object corresponding to theoperation of deactivating a noise cancellation function included in thevirtual object 1630 in FIG. 16).

In an embodiment, an operation of completely blocking noise may indicatean operation of performing a function of preventing the user fromhearing a real sound by performing a noise cancellation function on thereal sound. For example, the wearable electronic device 100 maydetermine an NC level (e.g., 0) corresponding to the operation ofcompletely blocking noise. For example, based on a user input (e.g., auser input for activating a virtual object (e.g., the virtual object 912or the virtual object 922 in FIG. 9)) to a virtual object (e.g., thevirtual object 912 or the virtual object 922 in FIG. 9) corresponding tothe operation of completely blocking noise, the wearable electronicdevice 100 may determine the NC level of an object (e.g., the firstobject 910 or the second object 920 in FIG. 9) corresponding to thevirtual object corresponding to the user input to be an NC level (e.g.,0) corresponding to the operation of completely blocking noise. Asanother example, referring to FIG. 15 or FIG. 16 to be described later,the wearable electronic device 100 may determine an NC level of thewearable electronic device 100 to be an NC level (e.g., 0) correspondingto the operation of completely blocking noise, based on a user input toa virtual object corresponding to the operation of completely blockingall surrounding noises (e.g., the virtual object 1540 in FIG. 15 or avirtual object corresponding to the operation of completely blockingnoise included in the virtual object 1630 in FIG. 16).

According to an embodiment, a volume adjustment operation may indicatean operation of adjusting the volume of a real sound that the userhears. For example, even when the relative volume of a real sound is100, the relative volume of the real sound that the user hears may beadjusted to 1 to 99 by performing noise canceling on the real sound. Forexample, the wearable electronic device 100 may determine an NC level(e.g., 1 to 99) corresponding to the volume adjustment operation. Forexample, based on a user input to a virtual object corresponding to thevolume adjustment operation (e.g., the virtual object 914 or the virtualobject 924 in FIG. 9), the wearable electronic device 100 may determinethe NC level of an object (e.g., the first object 910 or the secondobject 920 in FIG. 9) corresponding to a virtual object corresponding tothe user input to be an NC level (e.g., 1 to 99) corresponding to thevolume adjustment operation. As another example, referring to FIG. 15 orFIG. 16 to be described later, based on a user input to a virtual objectcorresponding to the volume adjustment operation for all surroundingnoises (e.g., the virtual object 1550 in FIG. 15 or a virtual objectcorresponding to the volume adjustment operation included in the virtualobject 1630 in FIG. 16), the wearable electronic device 100 maydetermine the NC level to be an NC level (e.g., 1 to 99) correspondingto the volume adjustment operation.

According to an embodiment, the hearing aid operation may indicate anoperation in order for the user to hear a sound having a volume greaterthan that of a real sound. For example, sound data corresponding to areal sound may be obtained through a microphone, and the obtained sounddata may be amplified and output to a speaker, so that the user can heara sound having a volume greater than that of the real sound. Forexample, the wearable electronic device 100 may determine an NC level(e.g., 101 to 150) corresponding to the hearing aid operation. Thismeans that the higher the NC level corresponding to the hearing aidoperation, the greater the volume of a sound heard by the user. Forexample, based on a user input to a virtual object (e.g., the virtualobject 914 or the virtual object 924 in FIG. 9) corresponding to thehearing aid operation, the wearable electronic device 100 may determinethe NC level of an object (e.g., the first object 910 or the secondobject 920 in FIG. 9) corresponding to the virtual object correspondingto the user input to be an NC level (e.g., 101 to 150) corresponding tothe hearing aid operation. Alternatively, for example, referring to FIG.15 or FIG. 16 to be described later, based on a user input to a virtualobject corresponding to the hearing aid operation for all surroundingnoises (e.g., the virtual object 1560 in FIG. 15 or a virtual objectcorresponding to the hearing aid operation included in the virtualobject 1630 in FIG. 16), the wearable electronic device 100 maydetermine the NC level of the wearable electronic device 100 to be an NClevel (e.g., 101 to 150) corresponding to the hearing aid operation.

FIG. 13 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments.

Referring to FIG. 13, in operation 1301, according to certainembodiments, the wearable electronic device 100 (e.g., the processor320) may determine an NC level. For example, the wearable electronicdevice 100, as described in operation 805 above, may determine an NClevel corresponding to at least one object located in the vicinity ofthe wearable electronic device 100 (e.g., the first object 910 or thesecond object 920 in FIG. 9, or surrounding objects identified based onsurrounding environment information obtained from an external device(e.g., the external electronic device 520 and/or the server 530))(hereinafter, surrounding objects of wearable electronic device 100). Asanother example, as will be described later with reference to FIG. 15and FIG. 16, the wearable electronic device 100 may determine one NClevel corresponding to all surrounding objects of the wearableelectronic device 100.

In operation 1303, according to certain embodiments, the wearableelectronic device 100 may transmit a signal including information aboutthe determined NC level to an external device (e.g., directly to theexternal wearable electronic device 410, or to the external wearableelectronic device 410 through the external electronic device 520). Forexample, the wearable electronic device 100 may transmit a signalincluding information on NC levels corresponding to surrounding objectsof the wearable electronic device 100 to the external wearableelectronic device 410. In this case, the external wearable electronicdevice 410 may perform operations corresponding to the surroundingobjects (e.g., the respective operations described with reference toFIG. 12), based on the NC levels corresponding to the surroundingobjects. For example, the wearable electronic device 100 may transmit afirst signal including information about a first NC level (e.g., 0)corresponding to the first object 910 and information about a second NClevel (e.g., 100) corresponding to the second object 920 to the externalwearable electronic device 410. For example, the first signal mayinclude information indicating objects corresponding to respective NClevels. In this case, based on the respective NC levels included in thefirst signal, the external wearable electronic device 410 may perform anoperation of deactivating a noise cancellation function on the firstobject 910 (e.g., an operation of performing a function that allows theuser to hear a real sound as it is from the first object 910), andperform an operation of completely blocking noise on the second object920 (e.g., an operation of performing a function of preventing the userfrom hearing a real sound from the second object 920 by performing anoise cancellation function on the real sound from the second object920). As another example, the wearable electronic device 100 maytransmit a signal including information about one NC level correspondingto all surrounding objects of the wearable electronic device 100 to theexternal wearable electronic device 410. In this case, the externalwearable electronic device 410 may perform a corresponding operation(e.g., one of the operations described in FIG. 12), based on one NClevel corresponding to all surrounding objects of the wearableelectronic device 100. For example, the wearable electronic device 100may transmit a second signal including information about one NC level(e.g., 50) corresponding to all surrounding objects of the wearableelectronic device 100 to the external wearable electronic device 410.For example, the second signal may include information indicating thatthe NC level included in the second signal corresponds to allsurrounding objects of the wearable electronic device 100. In this case,based on the information about the NC level (e.g., 50) included in thesecond signal, if the relative volume of a real sound obtained from thesurrounding objects of the wearable electronic device 100 is 100, theexternal wearable electronic device 410 may perform noise canceling onthe real sound to adjust the real sound heard by the user to 50.

FIG. 14 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments. FIG. 14 will bedescribed with reference to FIG. 3, FIG. 5, and FIG. 9.

Referring to FIG. 14, in operation 1401, according to certainembodiments, the wearable electronic device 100 (e.g., a processor(e.g., the processor 320 in FIG. 3) of the wearable electronic device100), based on at least one image input through at least one camera(e.g., the first camera 311, the second camera 312, and/or the thirdcamera 313 in FIG. 3), may identify at least one object (e.g., the firstobject 910 or the second object 920 in FIG. 9) located in the vicinityof the wearable electronic device 100. The at least one camera (e.g.,the first camera 311, the second camera 312, and/or the third camera 313in FIG. 3) may capture an image. The wearable electronic device 100through a processor 320 can identify at least one object located in thevicinity based at least in part by recognizing the object in the image.

In operation 1403, according to certain embodiments, based on firstsound data input through at least one microphone (e.g., the sound inputdevice 362 in FIG. 3) and/or second sound data obtained from an externaldevice (e.g., the external wearable electronic device 410 and/or theexternal electronic device 520) through at least one communicationmodule (e.g., the communication module 370 in FIG. 3), the wearableelectronic device 100 may identify sound information corresponding to atleast one object (e.g., the first object 910 or the second object 920 inFIG. 9) located in the vicinity of the wearable electronic device 100.For example, the wearable electronic device 100 through the processor320 may identify sound information attributable to the identified atleast one object from first sound data input through the at least onemicrophone (e.g., the sound input device 362 in FIG. 3).

In operation 1405, according to certain embodiments, the wearableelectronic device 100, based on the sound information, may control atleast one display module (e.g., the display module 350 in FIG. 3) todisplay at least one virtual object (e.g., the first virtual object 911and/or the second virtual object 921 in FIG. 9) corresponding to theidentified at least one object at a position corresponding to the atleast one object (e.g., the first object 910 or the second object 920 inFIG. 9).

In operation 1407, according to certain embodiments, the wearableelectronic device 100 may obtain a first user input associated with afirst virtual object (e.g., the first virtual object 911 in FIG. 9)among the at least one virtual object.

In operation 1409, according to certain embodiments, the wearableelectronic device 100, based on the first user input, may determine anNC (noise cancellation) level of a first object (e.g., the first object910 in FIG. 9) corresponding to the first virtual object (e.g., thefirst virtual object 911 in FIG. 9).

In operation 1411, according to certain embodiments, the wearableelectronic device 100 may control at least one communication module(e.g., the communication module 370 in FIG. 3) such that a signalincluding information about the determined NC level is transmitted tothe external device through the at least one communication module (e.g.,the communication module 370 in FIG. 3) (e.g., directly to the externalwearable electronic device 410, or to the external wearable electronicdevice 410 through the external electronic device 520).

In certain embodiments, the wearable electronic device 100 may adjustthe sound output signal to the sound output device 363 in lieu ofoperation 1411 based on the NC determination during operation 1409.

FIG. 15 is a diagram illustrating a method of operating an externalelectronic device, according to certain embodiments. FIG. 15 will bedescribed with reference to FIG. 5 and FIG. 9.

Referring to FIG. 15, according to certain embodiments, an externalelectronic device 520 (e.g., a processor of the external electronicdevice 520) may display a screen 1520 through a display module 1510 ofthe external electronic device 520. For example, the screen 1520 mayinclude an object 1530 corresponding to an operation of deactivating anoise cancellation function, an object 1540 corresponding to anoperation of completely blocking noise, an object 1550 corresponding toa volume adjustment operation, and/or an object 1560 corresponding to ahearing aid operation. For example, based on sound information of atleast one object located in the vicinity of the wearable electronicdevice 100 (e.g., the first object 910 or the second object 920 in FIG.9, or surrounding objects identified based on surrounding environmentinformation obtained from an external device (e.g., the externalelectronic device 520 and/or the server 530)), if the volume of anambient noise is greater than or equal to a configured first volume, oris less than or equal to a configured second volume, the wearableelectronic device 100 may transmit a signal to the external electronicdevice 520, cause the external electronic device 520 may display ascreen 1520.

The wearable electronic device 100 may determine an NC level of thewearable electronic device 100, based on a user input to the screen 1520displayed on the display module 1510 of the external electronic device520.

FIG. 16 is a diagram illustrating a method of operating a wearableelectronic device, according to certain embodiments. FIG. 16 will bedescribed with reference to FIG. 3.

Referring to FIG. 16, according to certain embodiments, the wearableelectronic device 100 (e.g., the processor 320) may display an augmentedreality screen 1600 through the display module 350. For example, theuser may directly recognize at least one object located in the vicinity(e.g., a first object 1610 corresponding to a TV set and/or a secondobject 1620 corresponding to an external speaker) with the user's eyeswhile wearing the wearable electronic device 100, and the user mayrecognize the augmented reality screen 1600 displayed through thedisplay module 350 of the wearable electronic device 100.

Referring to FIG. 16, according to certain embodiments, the wearableelectronic device 100 may display a virtual object 1630 on the augmentedreality screen 1600 through the display module 350. For example, thevirtual object 1630 may include a virtual object corresponding to anoperation of deactivating a noise cancellation function, a virtualobject corresponding to an operation of completely blocking noise, avirtual object corresponding to a volume adjustment operation, and/or avirtual object corresponding to a hearing aid operation. For example,based on sound information of at least one object located in thevicinity of the wearable electronic device 100, if the volume of anambient noise is greater than or equal to a configured first volume, oris less than or equal to a configured second volume, the wearableelectronic device 100 may display the virtual object 1630. For example,the wearable electronic device 100 may determine an NC level of thewearable electronic device 100, based on a user input to the virtualobject 1630.

It is noted that in certain embodiments, the wearable electronic device100 may have preset settings configuring the first volume and secondvolume that are in places that the user frequents. For example, when theuser is at their home, the wearable electronic device 100 may havesettings that pre-configure based on history, the first volume andsecond volume of previously encountered objects.

FIG. 17 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments. FIG. 17 will bedescribed with reference to FIG. 3 and FIG. 18. FIG. 18 is a diagramillustrating a method of operating a wearable electronic device.

It is noted that certain objects can be audio devices that have volumesettings. In certain embodiments, the wearable electronic device 100 candirectly control the volume of these devices through, for example, aBluetooth or Wireless Local Area Network connection.

Referring to FIG. 17, in operation 1701, according to certainembodiments, the wearable electronic device 100 (e.g., the processor320) may display a virtual object (e.g., a virtual object 1830)corresponding to a user input to an augmented reality screen 1800through the display module 350. For example, the wearable electronicdevice 100 may obtain a user input, and display, on the augmentedreality screen 1800, a virtual object 1830 corresponding to the userinput (e.g., a virtual object corresponding to a virtual wall or avirtual curtain) at a position corresponding to the obtained user input.For example, in the case where the user wearing the wearable electronicdevice 100 and located in the living room wishes to remove the sound ofan external speaker 1620 in order to hear the sound of a TV set 1810,the user may enter a user input (e.g., a user gaze input, a userutterance input, a user touch input, a user gesture input, or a userinput through a separate input device) for specifying a desired positionon the augmented reality screen 1800. In this case, the wearableelectronic device 100 may display the virtual object 1830 at a positioncorresponding to the user input.

In operation 1703, according to certain embodiments, the wearableelectronic device 100, based on the position at which the virtual object(e.g., the virtual object 1830) is displayed, may transmit a signal foradjusting the sound of a specific object (e.g., the second object 1620located beyond the virtual object 1830 from the wearable electronicdevice 100) to an external device (e.g., directly to the externalwearable electronic device 410, or to the external wearable electronicdevice 410 through the external electronic device 520). For example, thesignal for adjusting the sound of a specific object, which istransmitted from the wearable electronic device 100 to the externalwearable electronic device 410, may indicate a signal includinginformation on the display position of the virtual object 1830 and/orinformation (e.g., position information of the second object 1620)indicating a specific object (e.g., the second object 1620) locatedbeyond the virtual object 1830. In this case, the wearable electronicdevice 100 may transmit a signal for adjusting the sound of a specificobject (e.g., the second object 1620) to the external wearableelectronic device 410, and the external wearable electronic device 410,based on the information on the display position of the virtual object1830 and/or information (e.g., position information of the second object1620) indicating a specific object (e.g., the second object 1620)located beyond the virtual object 1830, which are included in the signalreceived from the wearable electronic device 100, may perform noisecanceling by generating the opposite wave signal of the wave of the realsound from the specific object (e.g., the second object 1620).

According to certain embodiments, based on a user input, the wearableelectronic device 100, in operation 1701 described above, may displaythe virtual object 1830 in the form of a virtual wall or a virtualcurtain corresponding to the user input. For example, in the case wherethe virtual object 1830 is displayed in the shape of a virtual wall or avirtual curtain, the wearable electronic device 100 may determine amethod of adjusting a sound of a specific object (e.g., the secondobject 1620 positioned beyond the virtual object 1830 from the wearableelectronic device 100), based on properties configured to correspond tothe virtual object 1830 (e.g., the thickness, transparency, physicalproperties (e.g., water, metal, or concrete material) and/or volume ofthe virtual wall (or virtual curtain) selected in response to a userinput). For example, if the physical property of the virtual object 1830displayed in response to a user input is water, the wearable electronicdevice 100 may determine a sound adjustment method of the second object1620, thereby providing an effect as if the user hears the sound underthe water. As another example, if the virtual object 1830 displayed inresponse to a user input is a virtual curtain, the wearable electronicdevice 100 may determine the degree of noise canceling for the sound ofthe second object 1620 to be lower than that in the case where thevirtual object 1830 is a virtual wall, thereby increasing the relativevolume of the sound heard by the user from the second object 1620compared to the case where the virtual object 1830 is a virtual wall.The method of determining an adjustment method of a sound from aspecific object is exemplary, and there is no limitation thereto.

FIG. 19 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments.

Referring to FIG. 19, in operation 1901, according to certainembodiments, the wearable electronic device 100 (e.g., the processor320) may identify the type of an application executed in the wearableelectronic device 100.

In operation 1903, according to certain embodiments, the wearableelectronic device 100 may determine an NC level of the wearableelectronic device 100, based on the type of the application executed inthe wearable electronic device 100. For example, when a movie watchingapplication is executed, the wearable electronic device 100 maydetermine the NC level of the wearable electronic device 100 to be an NClevel (e.g., 100) corresponding to the operation of completely blockingnoise. For example, when an augmented reality conference application isexecuted, the wearable electronic device 100 may determine the NC levelof the wearable electronic device 100 to be an NC level (e.g., 1 to 99)corresponding to the volume adjustment operation. For example, when ahearing aid application is executed, the wearable electronic device 100may determine the NC level of the wearable electronic device 100 to bean NC level (e.g., 101 to 150) corresponding to the hearing aidoperation. The type of the executed application and the NC levelcorresponding thereto are exemplary, and there is no limitation thereto.

FIG. 20 is a flowchart illustrating a method of operating a wearableelectronic device, according to certain embodiments. FIG. 20 will bedescribed with reference to FIG. 21. FIG. 21 is a diagram illustrating amethod of operating a wearable electronic device.

Referring to FIG. 20, in operation 2001, according to certainembodiments, the wearable electronic device 100 (e.g., the processor(e.g., the processor 320 in FIG. 3) of the wearable electronic device100) may obtain sound data input through at least one microphone (e.g.,the sound input device 362 in FIG. 3). For example, the wearableelectronic device 100 may obtain sound data corresponding to a soundfrom at least one object located in the vicinity of the wearableelectronic device 100 (e.g., a person 2130 located in the vicinity)and/or sound data corresponding to a sound from a user 2110 wearing thewearable electronic device 100. For example, referring to FIG. 21, thewearable electronic device 100 may obtain sound data corresponding to asound from the person 2130 who exists in the vicinity of the wearableelectronic device 100 and/or the user 2110 wearing the wearableelectronic device 100 during an augmented reality meeting.

In operation 2003, according to certain embodiments, the wearableelectronic device 100 may transmit sound data input through at least onemicrophone (e.g., the sound input device 362 in FIG. 3) to an externaldevice, and, at this time, transmission to an external device mayindicate, for example, that the wearable electronic device 100 maytransmit sound data input through at least one microphone (e.g., thesound input device 362 in FIG. 3) to an external wearable electronicdevice (e.g., the external wearable electronic device 410 in FIG. 5)worn by the user 2110 wearing the wearable electronic device 100 and/oran electronic device of another user (e.g., another user 2120) (e.g.,the wearable electronic device worn by another user (e.g., the same typeof electronic device as the external wearable electronic device 410 inFIG. 5), or the electronic device 2202, the electronic device 2204, orthe server 2208 in FIG. 22). For example, referring to FIG. 21, thewearable electronic device 100 may transmit sound data input through atleast one microphone (e.g., the sound input device 362 in FIG. 3) to anexternal device (e.g., an external wearable electronic device (e.g., theexternal wearable electronic device 410 in FIG. 4) worn by the user 2110wearing the wearable electronic device 100 and/or an electronic deviceof another user (e.g., another user 2120) (e.g., the wearable electronicdevice worn by another user (e.g., the same type of electronic device asthe external wearable electronic device 410 in FIG. 4), or theelectronic device 2202, the electronic device 2204, or the server 2208in FIG. 22)) during the augmented reality meeting. At this time, forexample, the wearable electronic device 100 may transmit, to theexternal device, only sound data corresponding to a sound from the user2110 wearing the wearable electronic device 100, among the sound datainput through at least one microphone of the wearable electronic device100 (e.g., the sound input device 362 in FIG. 3), instead oftransmitting sound data corresponding to a sound from the person 2130existing in the vicinity of the wearable electronic device 100 to theexternal device. Accordingly, the user 2110 wearing the wearableelectronic device 100 and/or another user 2120 who does not exist in thevicinity of the wearable electronic device 100 but participates in theaugmented reality meeting may hear some of the sound data input throughat least one microphone (e.g., the sound input device 362 in FIG. 3) ofthe wearable electronic device 100. As another example, it may beunderstood by those skilled in the art that the wearable electronicdevice 100 may transmit both the sound data corresponding to a soundfrom the person 2130 existing in the vicinity of the wearable electronicdevice 100 and the sound data corresponding to a sound from the user2110 wearing the wearable electronic device 100 to the external device(e.g., an external wearable electronic device (e.g., the externalwearable electronic device 410 in FIG. 4) worn by the user 2110 wearingthe wearable electronic device 100 and/or an electronic device ofanother user (e.g., another user 2120) (e.g., the wearable electronicdevice worn by another user (e.g., the same type of electronic device asthe external wearable electronic device 410 in FIG. 4), or theelectronic device 2202, the electronic device 2204, or the server 2208in FIG. 22)).

Hereinafter, at least one configuration of the electronic device will bedescribed in more detail with reference to FIG. 22. In FIG. 22, theelectronic device 2201 may indicate the external electronic device 520in FIG. 5. In FIG. 22, the electronic device 2202 may indicate thewearable electronic device 100 and/or the external wearable electronicdevice 410 in FIG. 5. In FIG. 22, the server 2208 may indicate theserver 530 in FIG. 5. Alternatively, the wearable electronic device 100,the external wearable electronic device 410, and/or the server 530 inFIG. 5 may be implemented as the same type of electronic device as theelectronic device 2201 in FIG. 22.

FIG. 22 is a block diagram illustrating an electronic device in anetwork environment according to certain embodiments. Referring to FIG.22, the electronic device 2201 in the network environment 2200 maycommunicate with an electronic device 2202 via a first network 2298(e.g., a short-range wireless communication network), or an electronicdevice 2204 or a server 2208 via a second network 2299 (e.g., along-range wireless communication network). According to an embodiment,the electronic device 2201 may communicate with the electronic device2204 via the server 2208. According to an embodiment, the electronicdevice 2201 may include a processor 2220, memory 2230, an input module2250, a sound output module 2255, a display module 2260, an audio module2270, a sensor module 2276, an interface 2277, a connecting terminal2278, a haptic module 2279, a camera module 2280, a power managementmodule 2288, a battery 2289, a communication module 2290, a subscriberidentification module (SIM) 2296, or an antenna module 2297. In someembodiments, at least one of the components (e.g., the connectingterminal 2278) may be omitted from the electronic device 2201, or one ormore other components may be added in the electronic device 2201. Insome embodiments, some of the components (e.g., the sensor module 2276,the camera module 2280, or the antenna module 2297) may be implementedas a single component (e.g., the display module 2260).

The processor 2220 may execute, for example, software (e.g., a program2240) to control at least one other component (e.g., a hardware orsoftware component) of the electronic device 2201 coupled with theprocessor 2220, and may perform various data processing or computation.According to one embodiment, as at least part of the data processing orcomputation, the processor 2220 may store a command or data receivedfrom another component (e.g., the sensor module 2276 or thecommunication module 2290) in volatile memory 2232, process the commandor the data stored in the volatile memory 2232, and store resulting datain non-volatile memory 2234. According to an embodiment, the processor2220 may include a main processor 2221 (e.g., a central processing unit(CPU) or an application processor (AP)), or an auxiliary processor 2223(e.g., a graphics processing unit (GPU), a neural processing unit (NPU),an image signal processor (ISP), a sensor hub processor, or acommunication processor (CP)) that is operable independently from, or inconjunction with, the main processor 2221. For example, when theelectronic device 2201 includes the main processor 2221 and theauxiliary processor 2223, the auxiliary processor 2223 may be adapted toconsume less power than the main processor 2221, or to be specific to aspecified function. The auxiliary processor 2223 may be implemented asseparate from, or as part of the main processor 2221.

The auxiliary processor 2223 may control, for example, at least some offunctions or states related to at least one component (e.g., the displaymodule 2260, the sensor module 2276, or the communication module 2290)among the components of the electronic device 2201, instead of the mainprocessor 2221 while the main processor 2221 is in an inactive (e.g.,sleep) state, or together with the main processor 2221 while the mainprocessor 2221 is in an active (e.g., executing an application) state.According to an embodiment, the auxiliary processor 2223 (e.g., an imagesignal processor or a communication processor) may be implemented aspart of another component (e.g., the camera module 2280 or thecommunication module 2290) functionally related to the auxiliaryprocessor 2223. According to an embodiment, the auxiliary processor 2223(e.g., the neural processing unit) may include a hardware structurespecified for artificial intelligence model processing. An artificialintelligence model may be generated by machine learning. Such learningmay be performed, e.g., by the electronic device 2201 where theartificial intelligence is performed or via a separate server (e.g., theserver 2208). Learning algorithms may include, but are not limited to,e.g., supervised learning, unsupervised learning, semi-supervisedlearning, or reinforcement learning. The artificial intelligence modelmay include a plurality of artificial neural network layers. Theartificial neural network may be a deep neural network (DNN), aconvolutional neural network (CNN), a recurrent neural network (RNN), arestricted boltzmann machine (RBM), a deep belief network (DBN), abidirectional recurrent deep neural network (BRDNN), deep Q-network or acombination of two or more thereof but is not limited thereto. Theartificial intelligence model may, additionally or alternatively,include a software structure other than the hardware structure.

The memory 2230 may store various data used by at least one component(e.g., the processor 2220 or the sensor module 2276) of the electronicdevice 2201. The various data may include, for example, software (e.g.,the program 2240) and input data or output data for a command relatedthereto. The memory 2230 may include the volatile memory 2232 or thenon-volatile memory 2234.

The program 2240 may be stored in the memory 2230 as software, and mayinclude, for example, an operating system (OS) 2242, middleware 2244, oran application 2246.

The input module 2250 may receive a command or data to be used byanother component (e.g., the processor 2220) of the electronic device2201, from the outside (e.g., a user) of the electronic device 2201. Theinput module 2250 may include, for example, a microphone, a mouse, akeyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The sound output module 2255 may output sound signals to the outside ofthe electronic device 2201. The sound output module 2255 may include,for example, a speaker or a receiver. The speaker may be used forgeneral purposes, such as playing multimedia or playing record. Thereceiver may be used for receiving incoming calls. According to anembodiment, the receiver may be implemented as separate from, or as partof the speaker.

The display module 2260 may visually provide information to the outside(e.g., a user) of the electronic device 2201. The display module 2260may include, for example, a display, a hologram device, or a projectorand control circuitry to control a corresponding one of the display,hologram device, and projector. According to an embodiment, the displaymodule 2260 may include a touch sensor adapted to detect a touch, or apressure sensor adapted to measure the intensity of force incurred bythe touch.

The audio module 2270 may convert a sound into an electrical signal andvice versa. According to an embodiment, the audio module 2270 may obtainthe sound via the input module 2250, or output the sound via the soundoutput module 2255 or an external electronic device (e.g., an electronicdevice 2202 (e.g., a speaker or a headphone)) directly or wirelesslycoupled with the electronic device 2201.

The sensor module 2276 may detect an operational state (e.g., power ortemperature) of the electronic device 2201 or an environmental state(e.g., a state of a user) external to the electronic device 2201, andthen generate an electrical signal or data value corresponding to thedetected state. According to an embodiment, the sensor module 2276 mayinclude, for example, a gesture sensor, a gyro sensor, an atmosphericpressure sensor, a magnetic sensor, an acceleration sensor, a gripsensor, a proximity sensor, a color sensor, an infrared (IR) sensor, abiometric sensor, a temperature sensor, a humidity sensor, or anilluminance sensor.

The interface 2277 may support one or more specified protocols to beused for the electronic device 2201 to be coupled with the externalelectronic device (e.g., the electronic device 2202) directly orwirelessly. According to an embodiment, the interface 2277 may include,for example, a high definition multimedia interface (HDMI), a universalserial bus (USB) interface, a secure digital (SD) card interface, or anaudio interface.

A connecting terminal 2278 may include a connector via which theelectronic device 2201 may be physically connected with the externalelectronic device (e.g., the electronic device 2202). According to anembodiment, the connecting terminal 2278 may include, for example, anHDMI connector, a USB connector, an SD card connector, or an audioconnector (e.g., a headphone connector).

The haptic module 2279 may convert an electrical signal into amechanical stimulus (e.g., a vibration or a movement) or electricalstimulus which may be recognized by a user via his tactile sensation orkinesthetic sensation. According to an embodiment, the haptic module2279 may include, for example, a motor, a piezoelectric element, or anelectric stimulator.

The camera module 2280 may capture a still image or moving images.According to an embodiment, the camera module 2280 may include one ormore lenses, image sensors, image signal processors, or flashes.

The power management module 2288 may manage power supplied to theelectronic device 2201. According to one embodiment, the powermanagement module 2288 may be implemented as at least part of, forexample, a power management integrated circuit (PMIC).

The battery 2289 may supply power to at least one component of theelectronic device 2201. According to an embodiment, the battery 2289 mayinclude, for example, a primary cell which is not rechargeable, asecondary cell which is rechargeable, or a fuel cell.

The communication module 2290 may support establishing a direct (e.g.,wired) communication channel or a wireless communication channel betweenthe electronic device 2201 and the external electronic device (e.g., theelectronic device 2202, the electronic device 2204, or the server 2208)and performing communication via the established communication channel.The communication module 2290 may include one or more communicationprocessors that are operable independently from the processor 2220(e.g., the application processor (AP)) and supports a direct (e.g.,wired) communication or a wireless communication. According to anembodiment, the communication module 2290 may include a wirelesscommunication module 2292 (e.g., a cellular communication module, ashort-range wireless communication module, or a global navigationsatellite system (GNSS) communication module) or a wired communicationmodule 2294 (e.g., a local area network (LAN) communication module or apower line communication (PLC) module). A corresponding one of thesecommunication modules may communicate with the external electronicdevice 2204 via the first network 2298 (e.g., a short-rangecommunication network, such as Bluetooth™, wireless-fidelity (Wi-Fi)direct, or infrared data association (IrDA)) or the second network 2299(e.g., a long-range communication network, such as a legacy cellularnetwork, a 5G network, a next-generation communication network, theInternet, or a computer network (e.g., LAN or wide area network (WAN)).These various types of communication modules may be implemented as asingle component (e.g., a single chip), or may be implemented as multicomponents (e.g., multi chips) separate from each other. The wirelesscommunication module 2292 may identify or authenticate the electronicdevice 2201 in a communication network, such as the first network 2298or the second network 2299, using subscriber information (e.g.,international mobile subscriber identity (IMSI)) stored in thesubscriber identification module 2296.

The wireless communication module 2292 may support a 5G network, after a4G network, and next-generation communication technology, e.g., newradio (NR) access technology. The NR access technology may supportenhanced mobile broadband (eMBB), massive machine type communications(mMTC), or ultra-reliable and low-latency communications (URLLC). Thewireless communication module 2292 may support a high-frequency band(e.g., the mmWave band) to achieve, e.g., a high data transmission rate.The wireless communication module 2292 may support various technologiesfor securing performance on a high-frequency band, such as, e.g.,beamforming, massive multiple-input and multiple-output (massive MIMO),full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, orlarge scale antenna. The wireless communication module 2292 may supportvarious requirements specified in the electronic device 2201, anexternal electronic device (e.g., the electronic device 2204), or anetwork system (e.g., the second network 2299). According to anembodiment, the wireless communication module 2292 may support a peakdata rate (e.g., 20 Gbps or more) for implementing eMBB, loss coverage(e.g., 164 dB or less) for implementing mMTC, or U-plane latency (e.g.,0.5 ms or less for each of downlink (DL) and uplink (UL), or a roundtrip of 1 ms or less) for implementing URLLC.

The antenna module 2297 may transmit or receive a signal or power to orfrom the outside (e.g., the external electronic device) of theelectronic device 2201. According to an embodiment, the antenna module2297 may include an antenna including a radiating element composed of aconductive material or a conductive pattern formed in or on a substrate(e.g., a printed circuit board (PCB)). According to an embodiment, theantenna module 2297 may include a plurality of antennas (e.g., arrayantennas). In such a case, at least one antenna appropriate for acommunication scheme used in the communication network, such as thefirst network 2298 or the second network 2299, may be selected, forexample, by the communication module 2290 from the plurality ofantennas. The signal or the power may then be transmitted or receivedbetween the communication module 2290 and the external electronic devicevia the selected at least one antenna. According to an embodiment,another component (e.g., a radio frequency integrated circuit (RFIC))other than the radiating element may be additionally formed as part ofthe antenna module 2297.

The antenna module 2297 may form a mmWave antenna module. According toan embodiment, the mmWave antenna module may include a printed circuitboard, an RFIC disposed on a first surface (e.g., the bottom surface) ofthe printed circuit board, or adjacent to the first surface and capableof supporting a designated high-frequency band (e.g., the mmWave band),and a plurality of antennas (e.g., array antennas) disposed on a secondsurface (e.g., the top or a side surface) of the printed circuit board,or adjacent to the second surface and capable of transmitting orreceiving signals of the designated high-frequency band.

At least some of the above-described components may be coupled mutuallyand communicate signals (e.g., commands or data) therebetween via aninter-peripheral communication scheme (e.g., a bus, general purposeinput and output (GPIO), serial peripheral interface (SPI), or mobileindustry processor interface (MIPI)).

According to an embodiment, commands or data may be transmitted orreceived between the electronic device 2201 and the external electronicdevice 2204 via the server 2208 coupled with the second network 2299.Each of the external electronic devices 2202 or 2204 may be a device ofa same type as, or a different type, from the electronic device 2201.According to an embodiment, all or some of operations to be executed atthe electronic device 2201 may be executed at one or more of theexternal electronic devices 2202, 2204, or 2208. For example, if theelectronic device 2201 should perform a function or a serviceautomatically, or in response to a request from a user or anotherdevice, the electronic device 2201, instead of, or in addition to,executing the function or the service, may request the one or moreexternal electronic devices to perform at least part of the function orthe service. The one or more external electronic devices receiving therequest may perform the at least part of the function or the servicerequested, or an additional function or an additional service related tothe request, and transfer an outcome of the performing to the electronicdevice 2201. The electronic device 2201 may provide the outcome, with orwithout further processing of the outcome, as at least part of a replyto the request. To that end, a cloud computing, distributed computing,mobile edge computing (MEC), or client-server computing technology maybe used, for example. The electronic device 2201 may provide ultralow-latency services using, e.g., distributed computing or mobile edgecomputing. In another embodiment, the external electronic device 2204may include an internet-of-things (IoT) device. The server 2208 may bean intelligent server using machine learning and/or a neural network.According to an embodiment, the external electronic device 2204 or theserver 2208 may be included in the second network 2299. The electronicdevice 2201 may be applied to intelligent services (e.g., smart home,smart city, smart car, or healthcare) based on 5G communicationtechnology or IoT-related technology.

According to certain embodiments, an electronic device 100 may include:at least one display module (e.g., the display module 350 in FIG. 3); atleast one communication module (e.g., the communication module 370 inFIG. 3); at least one microphone (e.g., the sound input device 390 inFIG. 3); at least one camera (e.g., the first camera 311, the secondcamera 312, and/or the third camera 313 in FIG. 3); and a processor 320,wherein the processor may be configured capture at least one imagethrough the at least one camera, to identify at least one object locatedaround the electronic device, based at least in part on the at least oneimage, identify sound information attributable to the identified atleast one object from first sound data input through the at least onemicrophone, control the at least one display module to display at leastone virtual object corresponding to the identified at least one objectat a position corresponding to the at least one object, wherein theposition is determined based on the sound information, obtain a firstuser input associated with a first virtual object among the at least onevirtual object, and determine a noise cancellation (NC) level of a firstobject corresponding to the first virtual object, based on the firstuser input.

According to certain embodiments, the processor may be configured toidentify the at least one object, based at least in part on surroundingenvironment information obtained through the at least one communicationmodule.

According to certain embodiments, the processor may be configured todetermine whether or not to display the first virtual object, based onvolume of sound included in first sound information corresponding to thefirst object.

According to certain embodiments, the processor may be configured toidentify a gaze direction of a user of the electronic device using theat least one camera, and determine whether or not to display the atleast one virtual object, based on the gaze direction.

According to certain embodiments, the processor may be configured toidentify at least a part of a body of a user of the electronic deviceusing the at least one camera, and determine whether or not to displaythe at least one virtual object, based on a position of the at least thepart of the body of the user.

According to certain embodiments, the processor may be configured tocontrol the at least one display module to display a second virtualobject (e.g., the virtual object 1830 in FIG. 18) corresponding to asecond user input through the at least one display module, and controlthe at least one communication module to transmit, to the externaldevice, a signal for adjusting a sound of a second object (e.g., thesecond object 1620 located beyond the virtual object 1830 from thewearable electronic device 100) among the at least one object, based ona position at which the second virtual object is displayed.

According to certain embodiments, the processor may be configured todetermine a method of adjusting a sound of the second object, based on apredetermined property corresponding to the second virtual object.

According to certain embodiments, the processor may be configured toidentify a type of an application executed in the electronic device anddetermine the NC level, based on the type of the application.

According to certain embodiments, the processor may be configured tocontrol the at least one communication module to transmit the firstsound data input through the at least one microphone to the externaldevice.

According to certain embodiments, the processor may be configured toreceive second sound data from an external device through the at leastone communication module and wherein identifying sound informationattributable to the identified at least one object from the first sounddata is based at least in part on the second sound data.

According to certain embodiments, the processor may be configured tocontrol the at least one communication module to transmit a signalincluding information about the determined NC level to the externaldevice through the at least one communication module.

According to certain embodiments, a method of operating an electronicdevice 100 may include capturing at least one image through at least onecamera of the electronic device (e.g., the first camera 311, the secondcamera 312, and/or the third camera 313 in FIG. 3), identifying at leastone object located around the electronic device, based at least in parton the at least one image, identifying sound information attributable tothe identified at least one object from first sound data input throughat least one microphone (e.g., the sound input device 390 in FIG. 3) ofthe electronic device, displaying at least one virtual objectcorresponding to the at least one object at a position corresponding tothe identified at least one object through at least one display module(e.g., the display module 350 in FIG. 3) of the electronic device,wherein the position is determined based on the sound information,obtaining a first user input associated with a first virtual objectamong the at least one virtual object, and determining a noisecancellation (NC) level of a first object corresponding to the firstvirtual object, based on the first user input.

According to certain embodiments, the identifying of the at least oneobject may include identifying the at least one object located aroundthe electronic device, based on the at least one image input through theat least one camera and surrounding environment information obtainedthrough the at least one communication module.

According to certain embodiments, the method may further includedetermining whether or not to display the first virtual object, based onvolume of sound included in first sound information corresponding to thefirst object.

According to certain embodiments, the method may further includeidentifying a gaze direction of a user of the electronic device usingthe at least one camera, and determining whether or not to display theat least one virtual object, based on the gaze direction.

According to certain embodiments, the method may further includeidentifying at least a part of a body of a user of the electronic deviceusing the at least one camera, and determining whether or not to displaythe at least one virtual object, based on a position of the at least thepart of the body of the user.

According to certain embodiments, the method may further includedisplaying a second virtual object (e.g., the virtual object 1830 inFIG. 18) corresponding to a second user input through the at least onedisplay module, and transmitting, to the external device, a signal foradjusting a sound of a second object (e.g., the second object 1620located beyond the virtual object 1830 from the wearable electronicdevice 100) among the at least one object through the at least onecommunication module, based on a position at which the second virtualobject is displayed.

According to certain embodiments, the method may further includedetermining a method of adjusting a sound of the second object, based ona predetermined property corresponding to the second virtual object.

According to certain embodiments, the method may further includeidentifying a type of an application executed in the electronic device,and determining the NC level, based on the type of the application.

According to certain embodiments, the method may further includetransmitting the first sound data input through the at least onemicrophone to the external device through the at least one communicationmodule.

According to certain embodiments, the method further comprises receivingsecond sound data from an external device through at least onecommunication module and wherein identifying sound informationattributable to the identified at least one object from the first sounddata is based at least in part on the second sound data.

The electronic device according to certain embodiments may be one ofvarious types of electronic devices. The electronic devices may include,for example, a portable communication device (e.g., a smartphone), acomputer device, a portable multimedia device, a portable medicaldevice, a camera, a wearable device, or a home appliance. According toan embodiment of the disclosure, the electronic devices are not limitedto those described above.

It should be appreciated that certain embodiments of the disclosure andthe terms used therein are not intended to limit the technologicalfeatures set forth herein to particular embodiments and include variouschanges, equivalents, or replacements for a corresponding embodiment.With regard to the description of the drawings, similar referencenumerals may be used to refer to similar or related elements. It is tobe understood that a singular form of a noun corresponding to an itemmay include one or more of the things, unless the relevant contextclearly indicates otherwise. As used herein, each of such phrases as “Aor B”, “at least one of A and B”, “at least one of A or B”, “A, B, orC”, “at least one of A, B, and C”, and “at least one of A, B, or C”, mayinclude any one of, or all possible combinations of the items enumeratedtogether in a corresponding one of the phrases. As used herein, suchterms as “1st” and “2nd”, or “first” and “second” may be used to simplydistinguish a corresponding component from another, and does not limitthe components in other aspect (e.g., importance or order). It is to beunderstood that if an element (e.g., a first element) is referred to,with or without the term “operatively” or “communicatively”, as “coupledwith”, “coupled to”, “connected with”, or “connected to” another element(e.g., a second element), it means that the element may be coupled withthe other element directly (e.g., wiredly), wirelessly, or via a thirdelement.

As used in connection with certain embodiments of the disclosure, theterm “module” may include a unit implemented in hardware, software, orfirmware, and may interchangeably be used with other terms, for example,“logic”, “logic block”, “part”, or “circuitry”. A module may be a singleintegral component, or a minimum unit or part thereof, adapted toperform one or more functions. For example, according to an embodiment,the module may be implemented in a form of an application-specificintegrated circuit (ASIC).

Various embodiments as set forth herein may be implemented as software(e.g., the program 2240) including one or more instructions that arestored in a storage medium (e.g., internal memory 2236 or externalmemory 2238) that is readable by a machine (e.g., the electronic device2201). For example, a processor (e.g., the processor 2220) of themachine (e.g., the electronic device 2201) may invoke at least one ofthe one or more instructions stored in the storage medium, and executeit. This allows the machine to be operated to perform at least onefunction according to the at least one instruction invoked. The one ormore instructions may include a code generated by a complier or a codeexecutable by an interpreter. The machine-readable storage medium may beprovided in the form of a non-transitory storage medium. Wherein, theterm “non-transitory” simply means that the storage medium is a tangibledevice, and does not include a signal (e.g., an electromagnetic wave),but this term does not differentiate between where data issemi-permanently stored in the storage medium and where the data istemporarily stored in the storage medium.

According to an embodiment, a method according to certain embodiments ofthe disclosure may be included and provided in a computer programproduct. The computer program product may be traded as a product betweena seller and a buyer. The computer program product may be distributed inthe form of a machine-readable storage medium (e.g., compact disc readonly memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded)online via an application store (e.g., PlayStore™), or between two userdevices (e.g., smart phones) directly. If distributed online, at leastpart of the computer program product may be temporarily generated or atleast temporarily stored in the machine-readable storage medium, such asmemory of the manufacturer's server, a server of the application store,or a relay server.

According to certain embodiments, each component (e.g., a module or aprogram) of the above-described components may include a single entityor multiple entities, and some of the multiple entities may beseparately disposed in different components. According to certainembodiments, one or more of the above-described components or operationsmay be omitted, or one or more other components or operations may beadded. Alternatively or additionally, a plurality of components (e.g.,modules or programs) may be integrated into a single component. In sucha case, the integrated component may still perform one or more functionsof each of the plurality of components in the same or similar manner asthey are performed by a corresponding one of the plurality of componentsbefore the integration. According to certain embodiments, operationsperformed by the module, the program, or another component may becarried out sequentially, in parallel, repeatedly, or heuristically, orone or more of the operations may be executed in a different order oromitted, or one or more other operations may be added.

What is claimed is:
 1. An electronic device comprising: at least onedisplay module; at least one communication module; at least onemicrophone; at least one camera; and a processor, wherein the processoris configured to: capture at least one image through the at least onecamera, identify at least one object located around the electronicdevice, based at least in part on the at least one image, identify soundinformation attributable to the identified at least one object fromfirst sound data input through the at least one microphone, control theat least one display module to display at least one virtual objectcorresponding to the identified at least one object at a positioncorresponding to the at least one object, wherein the position isdetermined based on the sound information, obtain a first user inputassociated with a first virtual object among the at least one virtualobject, and determine a noise cancellation (NC) level of a first objectcorresponding to the first virtual object, based on the first userinput.
 2. The electronic device of claim 1, wherein the processor isconfigured to identify the at least one object, based at least in parton surrounding environment information obtained through the at least onecommunication module.
 3. The electronic device of claim 1, wherein theprocessor is configured to determine whether or not to display the firstvirtual object, based on volume of sound included in first soundinformation corresponding to the first object.
 4. The electronic deviceof claim 1, wherein the processor is configured to identify a gazedirection of a user of the electronic device using the at least onecamera, and determine whether or not to display the at least one virtualobject, based on the gaze direction.
 5. The electronic device of claim1, wherein the processor is configured to identify at least a part of abody of a user of the electronic device using the at least one camera,and determine whether or not to display the at least one virtual object,based on a position of the at least the part of the body of the user. 6.The electronic device of claim 1, wherein the processor is configured tocontrol the at least one display module to display a second virtualobject corresponding to a second user input through the at least onedisplay module, and control the at least one communication module totransmit, to the external device, a signal for adjusting a sound of asecond object among the at least one object, based on a position atwhich the second virtual object is displayed.
 7. The electronic deviceof claim 6, wherein the processor is configured to determine a method ofadjusting a sound of the second object, based on a predeterminedproperty corresponding to the second virtual object.
 8. The electronicdevice of claim 1, wherein the processor is configured to identify atype of an application executed in the electronic device and determinethe NC level, based on the type of the application.
 9. The electronicdevice of claim 1, wherein the processor is configured to control the atleast one communication module to transmit the first sound data inputthrough the at least one microphone to an external device.
 10. Theelectronic device of claim 1, wherein the processor is configured toreceive second sound data from an external device through the at leastone communication module and wherein identifying sound informationattributable to the identified at least one object from the first sounddata is based at least in part on the second sound data.
 11. Theelectronic device of claim 10, wherein the processor is configured tocontrol the at least one communication module to transmit a signalincluding information about the determined NC level to the externaldevice through the at least one communication module.
 12. A method ofoperating an electronic device, the method comprising: capturing atleast one image through at least one camera of the electronic device,identifying at least one object located around the electronic device,based at least in part on the at least one image, identifying soundinformation attributable to the identified at least one object fromfirst sound data input through at least one microphone of the electronicdevice, displaying at least one virtual object corresponding to the atleast one object at a position corresponding to the identified at leastone object through at least one display module of the electronic device,wherein the position is determined based on the sound information,obtaining a first user input associated with a first virtual objectamong the at least one virtual object, and determining a noisecancellation (NC) level of a first object corresponding to the firstvirtual object, based on the first user input.
 13. The method of claim12, wherein the identifying of the at least one object comprisesidentifying the at least one object located around the electronicdevice, based at least in part on surrounding environment informationobtained through at least one communication module of the electronicdevice.
 14. The method of claim 12, further comprising determiningwhether or not to display the first virtual object, based on volume ofsound included in first sound information corresponding to the firstobject.
 15. The method of claim 12, further comprising identifying agaze direction of a user of the electronic device using the at least onecamera, and determining whether or not to display the at least onevirtual object, based on the gaze direction.
 16. The method of claim 12,further comprising identifying at least a part of a body of a user ofthe electronic device using the at least one camera, and determiningwhether or not to display the at least one virtual object, based on aposition of the at least the part of the body of the user.
 17. Themethod of claim 12, further comprising displaying a second virtualobject corresponding to second user input through the at least onedisplay module, and transmitting, to the external device, a signal foradjusting a sound of a second object among the at least one objectthrough the at least one communication module, based on a position atwhich the second virtual object is displayed.
 18. The method of claim17, further comprising determining a method of adjusting a sound of thesecond object, based on a predetermined property corresponding to thesecond virtual object.
 19. The method of claim 12, further comprisingidentifying a type of an application executed in the electronic device,and determining the NC level, based on the type of the application. 20.The method of claim 12, further comprising transmitting the first sounddata input through the at least one microphone to an external devicethrough at least one communication module of the electronic device.